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OUTLINE 


• Binomial Distribution 

• Process for determining observed probability of hit (POH) and 
associated confidence limits 


DOEPOD Software 
DOEPOD Analysis 
Future work 


Background 

"I'm too much of a knucklehead to know this stuff" 

"Oh boy, confidence limits. I hate these. " 

"They don't use 90/50 they use 90/95.” 

“I should have used 90/50” 

"I defer my answer to the statistician” 

"I'm not a statistician. ” 

"90/50 POD means that there is a 50% chance that the true POD is greater than 
90% at that flaw size?" Responses: "No. ", and "Yes. " , rest of world gives blank 
stares. 

“Confusion over ‘common definitions’ continues to be an issue...” 


“We have been using 29 out of 29 clandestinely for years” 


Background (continued) 
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• A core issue here is that the NDE personnel, nationwide, have different levels of 
understanding of statistics, and have delegated basic NDE POD statistical 
analysis to the statisticians. 

• This environment created a divergence in the interrelationship between the 
physics of the inspection procedure and the POD statistics. 

• NDE community should not blindly accept statistical results, but rather challenge 
the statistical results. 

• When NDE personnel defer explanations on statistical confidence bounds to 
others, it's like saying "I don't understand the error bars of my data". 

• This is not a good position. 

• We all need to learn and to speak the language of the other. 


This is the authors attempt to begin to bridge this gap. 


Using The Binomial Distribution 
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The binomial distribution describes the behavior of a count variable X if the following conditions 
apply: 

> The number of observations N is fixed. 

> Each observation is independent. 

> Each observation represents one of two outcomes ("success" or "failure"). 

> Use “green” or “red” to represent “Hit” or “Miss”, respectively. 

> The probability of ’’Hit" (POH) is the same for each outcome. 

If these conditions are met, then X has a binomial distribution. 


Set threshold 


Is it Green? 



ANALOGY 

Various shades 
(amplitudes) may 
be due to crack 
size, signal 
amplitude, etc. 

1 

Hit/Miss 


By setting a threshold only two outcomes (Hit/Miss) observations are obtained. 



Using The Binomial Distribution (continued) 
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Number of Hits, X, after N trials 


O = Example 
values of POH 
as observed 
during a test. 


Use binomial distribution for now. 

Other distributions may be used if they can be demonstrated to be better. 


Probability of Hit (POH) Example 


• Start with 61 flaws in the group. 

• Each flaw has with the same probability of being observed as a Hit. 

• Make 61 observations. 

• If 59 Hits are observed, then the Probability of Hit is POH = 59/61 = 0.97 (the observed 
frequency) 

• This is an estimated POH since the true POH can only be approached by making an 
“infinite” amount of observations. 

• Now that the Probability of Hit is measured, what is the confidence in that value? 

• This is somewhat analogous to asking what are the error bars or uncertainty in 
measurements. 

• Confidence level is the measure of probability associated with a confidence interval 
expressing the probability of truth of a statement that the interval will include the 
parameter value. 

• For NDE applications, the confidence bound of interest for Probability of Hit is the lower 
confidence bound. 


95% Confidence Level Example 

If we achieve a lower confidence limit (bound, value) = 0.90 , then 

“There is a 95% chance that the true POH is greater than 90% for that flaw size” 
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Parameter value = mean POH = <POH> 


Confidence Level Example (continued) 


With X = 59 Hits after N = 61 trials, yielding 59/61 = 0.97 POH (the observed frequency), the 
lower confidence bound, p , may be obtained from* 


X+(7V-X+l)F„W,/ 2 ) 


= 2.25 


\fi= 2(N-X + 1) = 6 

I /2 = 2X = 118 


P, = 0.9 

a is, a priori, the confidence level, 95%, that we are requiring 
FJfi.f,) is obtained from the F-distribution statistical table 


Note that the POH does NOT change if the confidence level is changed 


*lntroduction to Statistical Analysis”, W. J. Dixon and F. J. Massey Jr., 3 rd Edition, 1969, “Statistical Theory with Engineering Applications”, A. Hald, 
1952, and “Recommended Practice for Demonstration of Nondestructive (NDE) Reliability on Aircraft Production Parts”, Ward D. Rummel, Materials 
Evaluation, vol 40, August 1982 


95% Confidence Level Example (continued) 


P, = 0.9 


There is an 95% chance that the true POH is greater than 0.9 at that flaw size 

Or 

There is a 95% chance that the inspection system reliability is greater than 0.9 at that flaw size 

Or 

90/95 POD at that flaw size 


Directed Design of Experiments for Validating Probability of 
Detection of Inspection Systems (DOEPOD) 


DOEPOD Concepts 


> Not all flaws are created equally; never identical; but they may be grouped 
into classes by size, length, depth, etc. These classes have ranges or widths. 


>DOEPOD is a confidence value driven approach. 


> DOEPOD uses moving class width and variable class width optimization to 
identify the best lower confidence bound. Class widths start at 0.001” and 
increase. 


> DOEPOD uses real initial flaw, simulated, or completed inspection data sets. 
Guidelines are in the manual. (Smallest number of samples is 5). 


Grouping of Flawed Specimens 
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• Flaws may be grouped into classes by size, length, depth, etc. The grouping may 
be any class width, e.g., 0.001”, 0.036”, 0.100”, etc. 


• If there are sufficient number of flaws, then a moving “class width” may be used to 
dynamically group adjacent flaws into classes with widths (classwidths). 

> E.g., all flaws in the range 0.050” - 0.150” may be in a group, with the largest flaw being 
the identifier for the group. 

- Class width here is 0.100” 

- The next group may contain the range 0.049” - 0.149” ; the class width is moving 
from largest to smallest flaws. 


• POH needs to be determined for each flaw size grouping (number of flaws in each 
group is not necessarily the same) 


• Confidence bounds need to be determined for each flaw size grouping (both X and 
N vary for each group) 


What is the optimum class width? 
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Observed Probability of Hit (POH) 
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Effect of Grouping of 
Flawed Specimens 


• POH 0.001" Classwidth 
A PL 0.001" Classwidth 
X Hit/Miss 


Size (inch), XI 


Observed Probability of Hit (POH) 


Important: When grouping 
flaws, the observed POH and 
confidence bound is assigned 
to the largest flaw in the group 


_; 0 - 




Size (inch), XI 


X Hit/Miss 

• POH 0.100" Classwidth 
a PL 0.100 n Classwidth 


Anaty&iE file name: DCtPOD.vl Ai. G.PC.Jtli 
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DOEPOD 

DOEPOD Concepts (continued) 

> Using POH lower confidence bound (value) as driver for directing DOE. 

> If 90/95 POD is reached at a then identify locations that need additional 
validation for larger flaw sizes. 

> If 90/95 POD is not reached then use best lower confidence value to identify 
where options are available to reach 90/95 POD. 

> Identify CASE of the data set. 



> Provide directions, depending on the CASE, on how to modify the DOE to 
continue to efficiently validate the inspection system. 


> Determine false call rate and associated confidence limit 1 


DOEPOD Parameters 


Class length 

Inspection parameter (length, depth, etc.) 

Hit 

Flaw is detected 

Miss 

Flaw is not detected 

Need 

Add new samples to the existing set in order to reach the number of samples 
required at the class length 

LCL 

Lower confidence bound (value) of POH @ 95% confidence 

Opt. X P0H 

Optimum X P0H is identified for non-survey data sets. Optimum X P0H is the smallest class length 


Optimum X P0H and largest class width at which the minimum X POH = 1 occurs. Optimum X P0H may be more aggressive than 
optional, X PODopt , or X BestLCL , when the class width is constrained to the companion Optimum X POH class width listed. DOEPOD 


POH 

does not force use of Optimum X POH over X PODopt or X BestLCL Stability has not been demonstrated at Optimum X POH therefore 
there is an additional risk that can not be satisfied. 

Probability of Hit (Number of Hits in Classwidth/Total Number of Trials in Classwidth) 

POD 

Probability of Detection (the true POD obtained if an infinite number of samples are used) 

Signal 

Amplitude 

Scalar amplitude output of NDE inspection system 

Survey Data 
Sets 

Survey Data Sets are data sets that have a sparse or disperse collection of samples. The moving 

class width optimization has identified this set as having limited applications to moving class width processing. An alternate 
optimization of X P0H is used to provide guidance. 

Survey X P0H 

Survey X P0H is only identified for data sets determined to be Survey Data Sets. Survey X POH is the 
smallest class length and largest class width at which the minimum X P0H = 1 class length occurs. 

Survey X POH is the minimum class length at which X POD may be achieved when the class width is constrained to the companion 
survey class width listed. Survey X POH is utilized in all cases in which it occurs. 


DOEPOD Parameters (continued) 


^BestLCL Class length exhibiting the best LCL. The best class length is 
determined by increasing the moving class width until a 
maximum LCL is obtained 

Xj Class length X at point “i” 

X L Largest class length in entire data set 

X m Class length near the mid-point between the largest and the smallest 

class lengths having no misses 

X P0D Class length at which the lower confidence bound (value) is 0.90 or 
greater (90/95 POD) 

X P0H=1 Class length where there are no misses above this class length 

XpoDopt Recommended optional existing smaller class length where X P0D may 
also be achieved if additional sample are added. 

X s Smallest class length in data set 

UCL Upper confidence value of the false call rate @ 95% confidence 


Case #1 (Best Case) 


• 90/95 X P0D reached at a class length 

• Misses only below X P0D (and POH =1 everywhere greater than X P0D ) 

Directed Requirements for Validation of 90/95 X P0D 

^ Need samples at largest class length, X L 

> Need samples at mid-class length, X m « (X L - X P0D )/2 

> Option: Adding samples at X podopt may yield a new X P0D with a smaller class length. 

> If this is a survey data set, then only need to add samples at Survey X P0H (if listed) 

> Option: The user may add samples at Optimum X P0H The class width for all added samples at 
any class length is shown along with the Optimum X P0H . 

> The range of validation may be expanded by adding samples at 2 X L , 4X L , 8X L , 1 6X L ,etc , if 
the current range of validation is too small. 


Case #1 (Best Case) 


Detection Probability 
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File Name = 
Data Set Name - 
Date & lime - 
Xpod 90/95 Reached Anywhere? 
Classwidth @ 90/95 Xpod = 
Classlength @ 90/95 Xpod - 
Actual Lower Confidence Value - 
Best LCL = 
Classwidth @ Best LCL - 
Classlength @ Best LCL - 
a(1 ) [Alpha] = 
a(2) [Beta] = 
Chi-Square = 


A8002L.Case1.XLS 
A8002L.Case1(Eci-a-bf 

5/11/07 4:26 PM 
REACHED 


0.0040 

inch 

0.0147 

inch 

0.9120 

inch 

10.34 

inch 

1.94 


0.0023 



CASE 1 - 90/95 Xpod is VALIDATED from Xpod to XL 
vi ben XL andXm are satisfied. An alternate 90/95 Xpod is 
available ifXpodpot or Optimum Xpoh (if fisted) is also 
satisfied. 


Survey /Optimum Xpoh - 0.0115-0.001 inch (need 

NTlAC 90% POD * 0.942 @ 0.010 

NTTAC 90/95 POD - 0.952 @ 1015 


0.0000 


0.0500 


0.1000 


Note: Xpodopt is within one cfass width 
of Xpod r 


0.1500 0.2000 0.2500 

Size (inch), XI 

Analysis Hie name: DOE POD.vl .4.5.5.x Is 


0.3000 


0.3500 


0.4000 


False Call Rate = 

Largest Classlength , XL - 
Samples Needed @ XL - 
Classlength Mid-point , Xm = 
Samples Needed @ Xm = 
Smallest Classlength, Xs = 
Samples Needed @ Xs - 
New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl = 
Samples Needed @ Xlcl = 
POH Classlength, Xpoh = 
Samples Needed @ Xpoh - 
New Largest Classlength , 2XL = 
Xm is Near Verification Point - 
Opt. POD classlength, Xpodopt - 
Samples Needed ©Xpodopt = 


with UCL @ 95% = 
0.342 
28 

0.161 
26 


0.0145 

29 


16 Samples) 
inch 
inch 


inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Class Range A Actual Lower Confidence Value (95%, F-distributi on) 


Estimate of POH - Marquardt Gradient/Grid X Hit/Miss 





Case #2 
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• 90/95 X P0D reached at a class length 

• There are misses below X P0D and some misses above X P0D This is expected as X P0D nears 
the capability of the inspection system. 

• Since misses exist at class lengths X; above X P0D , then these greater lengths need to be 
validated, (i.e., The POH < 1 at class lengths X; above X P0D point, X P0D , so these greater 
lengths need to be validated.) 


(a) Add samples of class length X; where POH<1 (TABLE A). Starting from largest 

class length, X; and work toward small class lengths until reaching an acceptable X P0D or 
reaching X P0D 

(b) Add samples of class length X; where POH=1 (TABLE B). Accept a larger X POD class length at 
any of the X; . This acceptance is valid as long as any existing larger class lengths where POH<1 
are shown [via (a) above] to be at 90/95 X P0D or greater. Acceptance of a larger X POD is not 
necessarily the X P0D capability for the inspection system. 

In summary, satisfy the smallest X P0D in Table B that is greater than the largest X P0D in Table A, 
and/or the largest X POD in Table A. 


> If this is a survey data set, then only need samples at Survey X POH (if listed), rather than at X P0H=1 

> Option: If Optimum X POH < X POH=1 then the user may add samples at Optimum X POH rather than at X P0H=1 
The class width for all added samples at any class length is shown along with the Optimum X POH 


Directed Requirements 


> There are two (2) options that may be used to move this Case #2 toward Case #1 


> Need samples at largest class length, X L 


Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 


Case #2 


Detection Probability 



File Name - 
Data Set Name - 
Date &Time - 
Xpod 90/95 Reached Anywhere? 
Classwidth g 90/95 Xpod - 
Classlength g 90/95 Xpod - 
Actual Lower Confidence Value - 
Best LCL = 
Classwidth g Best LCL - 
Classlength g Best LCL - 
a(1) [Alpha] - 
a(2) [Beta] = 
Chi-Square - 


A8003L.Case2.XLS 

A8003L.Case2(Eci-a-bfi 

5/11/07 4:27 PM 

REACHED 
0,0040 inch 
0,0147 inch 
0,9120 



39.07 

8.21 

0.0035 


CASE 2 - 90/95 Xpod is reached at a class length. 

Further VAUDATiON is required. Recommend satisfying 
XL and the smallest Xpod in TABLE B that is greater than 
the largest Xpod in TABLE A t and/or the largest Xpod in 
Table A. 


Survey /Optimum Xpoh = 

NTIAC 90% POD = 

NTlAC 90/95 POD = 

False Call Rate = 

Largest Classlength s XL = 
Samples Needed g XL - 
Classlength Mid-point 5 Xm = 
Samples Needed g Xm = 
Smallest Classlength, Xs - 
Samples Needed g Xs = 
New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl = 
Samples Needed g Xlcl - 
POH Classlength, Xpoh - 
Samples Needed g Xpoh - 
New Largest Classlength , 2XL = 
Xm is Near Verification Point - 
Opt. POD classlength, Xpodopt = 
Samples Needed gXpodopt - 


0.000 inch (need 
0.906 @ 0.010 

0.933 @ 3.015 

with UCL @ 95% - 
0.342 


28 


Samples) 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Glass Range A Actual Lower Confidence Value (95%, F-distribution) 


Estimate of POH - Marquardt Gradient/Grid X 


Hit/Miss 




Number of Additional Samples Needed 


Case #2 

CASE 2 - 90/95 Xpod is reached at a class length. 

Further VALIDATION is required. 

Recommend satisfying XL and the smallest Xpod in TABLE B that is greater than the largest Xpod in TABLE A, and/or 
the largest Xpod in Table A. 


File Name = A8003L.Case2.XLS 
Data Set Name = A0OG3L,Case2[Eci-a-b0) 

Directed DOE Options 


TABLE C 

Class Length (in) Additional Samples 



XL = 0.342 28 


Xm = 
Xs = 
Xss = 
Xlcl = 
Xpoh - 
2XL - 
^Alternate Xm = 
Xpodopt - 


TABLE A* 

Selected class lengths 
with existing misses. 
Each point requires 
additional samples In or 
to achieve the Xpod 
listed . 


Xpod . Class 

No. 

Length (in) 

Need 

0.1861 

44 

0.0437 

32 

0.0429 

26 

0.0421 

25 


TABLE B* 

Selected class 
lengths with no 
misses. Additional 
samples at these 
class lengths vail 
achieve the Xpod 
listed. 


Xpod. Class 

No. 

Length (in) 

Need 

0.1992 

28 

0.0484 

26 

0.0464 

24 


Class Length (inch) 


O No Misses Observed ■ At Least One Miss OccuredAXL OXm OXs +Xss XXIcl XXpoh A2XL XXpod ♦Xpodopt 


Case #4 


23 


• 90/95 X P0D Not Reached 

• Best LCL is below 0.9 for the best class width group 

• No misses at or greater than class length exhibiting the best LCL, 
X Be st i_ci_(i- e -’ POH = 1 everywhere at or greater than class length 
currently having the best LCL, X BestLCL ) 


Directed Requirements 

y Need samples of X Best LCL in class length to achieve 90/95 X P0D at X Best LCL . X Best LCL 
may equal X L or X P0H =i so that the number of samples listed at this class length are 
redundantly the same and only one set of samples is needed. 

> If this is a survey data set, then need to add samples at Survey X P0H (if listed), rather 
than at X P0H=1 

> Option: If Optimum X P0H < X P0H =i then the user may add samples at Optimum X P0H 
rather than at X P0H =i. The class width for all added samples at any class length is shown 
along with the Optimum X P0H . 


Case #4 


Detection Probability 
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File Name - 

C7002L.CASE4.XLS 

Data Set Name - 

C7002L.CASE4(Lpi-a) 

Date & Time = 

6/6/07 10:56 AM 

Xpod 90/95 Reached Anywhere? 

NOT REACHED 

Classwidth @ 90/95 Xpod = 

inch 

Classlength @ 90/95 Xpod - 
Actual Lower Confidence Value - 

■ inch 

BestLCL- 

0.3444 

Classwidth @ Best LCL = 

0.0970 inch 

Classlength @ Best LCL - 

0.2131 inch 

a(1) [Alpha] = 

24.24 

a(2) [Beta] = 

13.26 

Chi-Square = 

0.0007 


CASE 4 - 90/95 Xpod is not reached anywhere. 
Recommend satisfying XL and the greater of Xpoh or 
Xlcl. 


0.0500 


0.1000 


0.1500 0.2000 0.2500 

Size (inch), XI 

Analysis file name: DGEFOD.vi .4.5.6.xls 


0.3000 


0.3500 


0.4000 


Survey/Optimum >t>oh - 0.0976 -0.004 inch (need 

NTIAC 90% POD - 0-001 @ 0.350 

NTtAC 90/95 POD = 0.901 @ 1580 

False Call Rate = with U CL @ 95% = 

Largest Classlength , XL = 0.342 

Samples Needed @ XL = 27 

Classlength Mid-point , Xm = 

Samples Needed @ Xm = 

Smallest Classlength, Xs = 

Samples Needed @ Xs - 
New Smaller Classlength, Xss = 

BestLCL Classlength, Xlcl = 

Samples Needed @ Xlcl - 
POH Classlength, Xpoh = 

Samples Needed @ Xpoh = 

New Largest Classlength , 2XL - 
Xm is Near Verification Point - 
Opt. POD classlength, Xpodopt - 
Samples Needed (gXpodopt = 


0.213 

11 

0.199 

13 


23 Samples) 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Class Range A Actual Lower Confidence Value (95%, F-distribution) 


Estimate of POH - Marquardt Gradient/Grid X Hit/Miss 


Case #5 

• 90/95 X P0D Not Reached 

• Best LCL is below 0.9 for the best class width group 

• There are misses at or greater than class length X BestLCL 

• There exists a class length, X P o H=1 , above which there are no 
misses. 

• There are no misses for class lengths equal to greater than X L /3 
(i.e., XpoH=i 5 X l /3) 

• Xp 0H =i < X l /3 so that POH is not fluctuating at larger class lengths. 
Use Xp 0H =i as the trial X P0D 

Directed Requirements 


> Need samples of X P0H =i 

> If this is a survey data set, then need to add samples at Survey X P0H (if listed), rather 
than at X P qi _|— ^ 

> Option: If Optimum X P0H < X P0H =i ^ en user ma y a ^d samples at Optimum X P0H 
rather than at X P0H=1 The class width for all added samples at any class length is shown 
along with the Optimum X P0H 

> Need 29 samples largest class length, X L 


Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 



Case #5 


26 


Detection Probability 
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File Name - 
Data Set Name - 
Date &Time - 
Xpod 90/95 Reached Anywhere? 
Classwidth @ 90/95 Xpod - 
Classlength @ 90/95 Xpod = 
Actual Lower Confidence Value = 
Best LCL = 
Classwidth @ Best LCL - 
Classlength @ Best LCL - 
a(1) [Alpha] = 
a(2) [Beta] - 
Chi-Square - 


CASE 5 - 90/95 Xpod is not reached anywhere. 
Recommend satisfying XL and Xpoh. 


C7001L.Case5.XLS 
C7001 L.Case5(Lpi-a) 

5/11/07 4:31 PM 
NOT REACHED 



inch 

inch 

0.6493 

0.0040 

inch 

0.0736 

inch 

14.03 


4.74 


0.0066 



Survey/Optimum )^>oh - 0.0BB1 -0.004 inch 

NTIAC 90% POD = 0.903 $ 

NTIAC 90/95 POD = 0.901 i 


(need 26 Samples) 
0.215 j nc h 
' 1360 inch 


0.0500 


0.1000 


-> Optimum Xpoh Available; Using Best 
LCL 


0.1500 0.2000 0.2500 

Size (inch), XI 

Analysis file name: DQEPQD.vl A5.5.xls 


0.3000 


0.3500 


0.4000 


False Call Rate = with U CL @ 

Largest Classlength , XL - 
Samples Needed @ XL = 
Classlength Mid-point , Xm - 
Samples Needed @ Xm = 
Smallest Classlength, Xs = 
Samples Needed @ Xs - 
New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl - 
Samples Needed @ Xlcl - 
POH Classlength, Xpoh = 
Samples Needed @ Xpoh = 
New Largest Classlength , 2XL - 
Xm is Near Verification Point - 
Opt. POD classlength, Xpodopt = 
Samples Needed @Xpodopt = 


! 95% = 
0.342 
26 


0.066 

26 


inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Class Range 4 Actual Lower Confidence Value (95%, F-distribution) 


Estimate of POH - Marquardt Gradient/Grid X Hit/Miss 


Case #6 

• 90/95 X P0D Not Reached 

• Best LCL is below 0.9 for the best class width group 

• There are misses at or greater than class length X BestLCL 

• There exists a class length, Xp 0H =i, above which there are no misses. 

• There are are misses for class lengths greater than X L /3 (i.e., X P0H=1 > X L /3) 

• Xpoh=i - so that POH may be fluctuating rapidly. 
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Directed Requirements 

> Need to expand current range of X L by adding new samples with class lengths of 2X L or greater 

> Need samples of X P0H =i- X P0H =i may equal X L so that the number of samples at this class length are 
redundantly the same and only one set of samples is needed. 

> Need 29 samples at largest class length, 2X L 

> If this is a survey data set, then need to add samples at Survey X P0H (if listed), rather than at X P0H =i 

> Option: If Optimum X P0H < X P0H=1 then the user may add samples at Optimum X P0H rather than at 
XpoH=i. The class width for all added samples at any class length is shown along with the Optimum X P0H . 


Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 


Case #6 


Detection Probability (Utilization of DGEPOD results requires approval of Engineering Authority) 



Size (inch), XI 

Analysis file name: DOE P OD. vl .4.5. 6.xls Beta. 3 


File Name - 

B1003AD.Case6.XLS 

Data Set Name - 

B1 0G3AD.C ase6(CRK 

Date & Time - 

8/6/07 3:22 PM 

Xpod 90/95 Reached Anywhere? 

NOT REACHED 

Classwidth @ 90/95 Xpod = 

inch 

Classlength @ 90/95 Xpod - 
Actual Lower Confidence Value = 

inch 

Best LCL = 

0,8514 

Classwidth @ Best LCL - 

0.0060 inch 

Classlength @ Best LCL = 

0.0603 inch 

a(1) [Alpha] = 

4.03 

a(2) [Beta] = 

0.99 

Chi-Square - 

0.0743 


CASE 6 - 90/95 Xpod is not reached anywhere. 


; Recommend satisfying XL, Xpoh, and 2XL. 

* 

* 

* 

* 

* 

Survey/ Optimum Xpoh - 0,1503 -0.002 inch (need 

28 Samples) 

NTIAC 90% POD- @ 

inch 

NTfAC 90/95 POD = @ 

inch 

False Call Rate - with U CL @ 95% = 

Largest Classlength , XL = 0-210 

inch 

Samples Needed @ XL - 28 

Classlength Mid-point , Xm = 

inch 

Samples Needed @ Xm - 

Smallest Classlength, Xs = 

inch 

Samples Needed @ Xs = 

New Smaller Classlength, Xss - 

inch 

BestLCL Classlength, Xlcl = 

inch 

Samples Needed @ Xlcl - 

POH Classlength, Xpoh = 0156 

inch 

Samples Needed @ Xpoh - 26 

New Largest Classlength , 2XL - 0 450 

inch 

Xm is Near Verification Points 

inch 

Opt, POD classlength, Xpodopt = 

inch 

Samples Needed cgXpodopt = 


O Probability of Hit (POH) in Class Range A Actual Lower Confidence Value (95%, F-distri button) 


Estimate of POH - Marquardt Gradient/Grid X Hit/Miss 


Case #7 
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• 90/95 X P0D Not Reached 

• Best LCL is below 0.9 for the best class width group 

• There are misses at or greater than class length X BestLCL 

• There does NOT exist a class length, X PO h=i> above which there are no 
misses. 

• POH may be fluctuating rapidly 

• There may be no hits anywhere 

Directed Requirements 


> Inspection system may not be appropriate for meeting inspection criteria 

> If this is a survey data set, then need to add samples at Survey X P0H (if listed) 

> Option: The user may add samples at Optimum X P0H . The class width for all added samples at any 
class length is shown along with the Optimum X P0H . 

> Need to expand current range of X L by adding new samples with class lengths of 2X L or greater 

> Need 29 samples at largest class length, 2X L 


Case #7 


Detection Probability 
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File Name - 

AB003(3)L. Case7.xls 

Data Set Name - 

AB003(3)L.Case7(CK. 1 

Date & Time - 

5/11/07 4:29 PM 

Xpod 90/95 Reached Anywhere? 

NOT REACHED 

Classwidth @ 90/95 Xpod - 

inch 

Classlength @ 90/95 Xpod - 
Actual Lower Confidence Value - 

■ inch 

BestLCL- 

0.7206 

Classwidth @ Best LCL - 

0.0060 inch 

Classlength @ Best LCL - 

0.1000 inch 

a(1) [Alpha] - 

1.07 

a(2) [Beta] - 

0.97 

Chi-Square - 

0.1997 


CASE 7 - 90/95 Xpod is not reached anywhere. 
Recommend satisfying 2XL and Optimum Xpoh (if 
iiisted). 


Size (inch), XI 

Analysis file name: DGEPGD.vl A5.5.xls 


Survey /Optimum Xpoh - 0.000 inch (need 

NT1AC 90% POD - S 

NTtAC 90/95 POD s @ 

False Cell Rate = with UCL @ 95% = 

Largest Classlength , XL - 
Samples Needed @ XL = 
Classlength Mid-point , Xm - 
Samples Needed @ Xm = 

Smallest Classlength, Xs - 
Samples Needed @ Xs = 

New Smaller Classlength, Xss = 

BestLCL Classlength, Xlcl - 
Samples Needed @ Xlcl - 
POH Classlength, Xpoh - 
Samples Needed @ Xpoh = 

New Largest Classlength , 2XL - 
Xm is Near Verification Point = 

Opt. POD classlength, Xpodopt - 
Samples Needed @Xpodopt - 


2.376 



Samples) 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

Inch 


O Probability of Hit (POH) in Class Range A Actual Lower Confidence Value (95%, F-distributi on) 


Estimate of POH - Marquardt Gradient/Grid X Hit/Miss 


Survey and Optimized X POH Data Sets 
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• This data set has insufficient number of samples for unconstrained class width 
optimization 

• The class width optimization has determined that there is a class width for which 
the smallest X P0H=1 class length is identified. The Survey and Optimum X P0H class 
lengths and class widths are identified on the charts as Survey/Optimum X P0H - 

> For example, the listing: 

Survey/Optimum X P0H = 0.0500 - 0.015 inch (need 18 samples) 

indicates that a class width of 0.015” has been used and the Survey or 
Optimum X P0H occurs at 0.0500”, and that 18 additional samples may be added 
to achieve X P0D add at that class length. The added samples should have sizes 
that range anywhere between 0.0500” and 0.035”, inclusively. 

• If Survey/Optimum X P0H is pursued, then the class width for all added samples at 
any other class length is to be the same as that for Survey/Optimum X P0H . 

Directed Requirements 

> Need samples at Survey/Optimum X P0H 

> Need samples at X L 

> Survey results are not available when user sets the class width 


Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 


Case #5 - Survey 


Detection Probability 
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File Name - 
Data Set Name - 
Date & Time - 
Xpod 90/95 Reached Anywhere? 
Classwidth @ 90/95 Xpod - 
Classlength @ 90/95 Xpod - 
Actual Lower Confidence Value - 
Best LCL = 
Classwidth @ Best LCL - 
Classlength @ Best LCL = 
a(1) [Alpha] - 
a(2) [Beta] - 
Chi-Square = 


Survey.Case5.xls 

Survey.Case5(Survey 


5/14/07 9:23 AM 

NOT REACHED 
«gggggg 5 inch 

i n r h 

0.2236 

0.0390 

inch 

0.4000 

inch 

5.36 


4.65 


0.0296 



CASE 5 - This is a survey data set 90/95 Xpod is not 
reached anywhere. Recommend satisfying XL and 
Survey Xpoh (if listed) 


Survey/Optimum Xpoh - 0.4600 -0.039 inch (need 26 Samples) 

NTIAC 90% POD “ @ inch 

NTIAC 90/95 POD = @ inch 


False Cali Rate = 


with UCL @ 95% = 
3.000 
26 


-> Survey Data Set; System Set Class 
Width 


0.5000 1.0000 1.5000 2.0000 

Size (inch), XI 

Analysis file name: DGEPGD.vL4.5.6.xls 


2.5000 


3.0000 


3.5000 


Largest Classlength , XL - 
Samples Needed @ XL = 
Classlength Mid-point , Xm - 
Samples Needed @ Xm - 
Smallest Classlength, Xs = 
Samples Needed @ Xs - 
New Smaller Classlength, Xss = 
BestLCL Classlength, Xlcl = 
Samples Needed @ Xlcl - 
POH Classlength, Xpoh - 
Samples Needed @ Xpoh - 
New Largest Classlength , 2XL = 
Xm is Near Verification Point = 
Opt. POD classlength, Xpodopt = 
Samples Needed @Xpodopt - 


0.460 

26 


inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Class Range A Actual Lower Confidence Value (95%, F-distri bution) 


Estimate of POH - Marquardt Gradient/Grid x Hit/Miss 


Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 
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Does the fastener 


DOEPOD Example 

type affect the capability of the 


inspection system? 


Detection Probability 



Size (inch), XI 

Analysis file name: DOEPOD. v1.4.5.6.xls 


File Name = 
Data Set Name - 
Date &Tme - 
Xpod 90/95 Reached Anywhere? 
Classwidth @ 90/95 Xpod - 
Classlength @ 90/95 Xpod - 
Actual Lower Confidence Value - 
Best LCL - 
Classwidth @ Best LCL = 
Classlength @ Best LCL = 
a(1) [Alpha] = 
a(2) [Beta] = 
Chi-Square - 


6/5/07 8:10 AM 

REACHED 
0.0380 inch 

0,1000 inch 
0.9070 



23.11 

6.34 

0.0000 


inch 

inch 


CASE 1 - 90/95 Xpod is VALIDATED from Xpod to XL 
wften XL and Xm are satisfied. An alternate 90/95 Xpod is 
available if Xpodpot or Optimum Xpoh (if listed) is also 
satisfied. 


Survey/Optimum Xpoh = 0.0620 -0.036 inch ['need 22 Samples) 


NTlAC 90% POD - 
NTIAC 90/95 POD = 

False Call Rate = 

Largest Classlength , XL - 
Samples Needed @ XL = 
Classlength Mid point , Xm - 
Samples Needed @ Xm = 
Smallest Classlength, Xs = 
Samples Needed @ Xs = 
New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl = 
Samples Needed @ Xlcl = 
POH Classlength, Xpoh - 
Samples Needed @ Xpoh - 
New Largest Classlength , 2XL = 
Xm is Near Verification Point = 
Opt. POD classlength, Xpodopt - 
Samples Needed @Xpodopt - 


with UCL @ 95% = 
0.200 
15 

0.125 
5 


0.0620 

22 


inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Class Range A Actual Lower Confidence Value (95%, F -distribution) 


Estimate of POH - Marquardt Gradient/Grid X 


Hit/Miss 


Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 


DOEPOD Example (continued) 
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Detection Probability 



Size (inch), XI 

Analysis file name: DOEPOD. vl .4. 5.6.xls 


File Name = 
Data Set Name - 
Date & Time = 
Xpod 90/95 Reached Anywhere? 
Classwidth @ 90/95 Xpod - 
Classlength @ 90/95 Xpod = 
Actual Lower Confidence Value - 
Best LCL- 
Classwidth @ Best LCL - 
Classlength @ Best LCL - 
a(1) [Alpha] = 
a(2) [Beta] = 
Chi-Square - 


6/5/07 8:22 AM 

NOT REACHED 
inch 
inch 


0.6146 


0.0300 

inch 

0.1800 

inch 

2.41 


0.85 


0.0157 



CASE 7 - 90/95 Xpod is not reached anywhere. 
Recommend satisfying 2XL and Optimum Xpoh (if 
ilisted). 


Survey/Optimum Xpoh = 0.000 inch (need Samples) 


NTIAC 90% POD - @ 

NTiAC mm POD = ® 


False Call Rate = with UCL @ 95% ^ 


Largest Classlength , XL - 
Samples Needed @ XL - 
Classlength Midpoint , Xm = 
Samples Needed @ Xm - 
Smallest Classlength, Xs - 
Samples Needed @ Xs - 
New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl - 
Samples Needed @ Xlcl - 
POH Classlength, Xpoh = 
Samples Needed @ Xpoh - 
New Largest Classlength , 2XL - 
Xm is Near Verification Point - 
Opt. POD classlength, Xpodopt - 
Samples Needed cgXpodopt - 


0.400 


inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) in Class Range 4 Actual Lower Confidence Value (95%, F-distribution) 


Estimate of POH - Marquardt Gradient/Grid 


X Hit/Miss 


DOEPOD Example (continued) 

Does the fastener type affect the capability of the inspection system? 


Optimum Xpoh (Best 
Xpod that could be 
achieved if test 


Multi- 

Fasteners Inspector 

90/95 Xpod (in) 

specimens are added) 
(in) 

DOEPOD 
vl .4.5.6 

5-Jun-07 

1 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

2 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

3 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

4 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

5 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

6 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

7 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

8 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

9 

0.100 

0.062 

CASE 1 

90/95 Xpod reached 

10 

0.150 

0.090 

CASE 1 

90/95 Xpod reached 

11 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

12 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

13 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

14 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

15 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

16 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

17 

0.200 

0.100 

CASE 1 

90/95 Xpod reached 

18 

0.200 

0.125 

CASE 1 

90/95 Xpod reached 

19 

0.200 

0.125 

CASE 1 

90/95 Xpod reached 





90/95 Xpod reached but 

20 

0.100 

see table A 

CASE 2 

miss at largest flaw size 

21 

not reached 


CASE 7 

Miss at largest flaw size 

22 

not reached 


CASE 7 

Miss at largest flaw size 



Probability of Hit (POH), Estimate of Probability of Hit (POH), 
Lower Confidence Limit, LCL 


36 



DOEPOD Example (continued) 
All Inspectors 


Detection Probability 



Size (inch), XI 

Analysis file name: DOEPOD. vl A 5.6.x Is 


File Name = 
Data Set Name - 
Date &Time - 
Xpod 90/95 Reached Anywhere? 
Classwidth @ 90/95 Xpod - 
Classlength @ 90/95 Xpod - 
Actual Lower Confidence Value = 
Best LCL - 
Classwidth @ Best LCL = 
Classlength @ Best LCL - 
a(1) [Alpha] = 
a (2) [Beta] = 
Chi-Square - 


5/31/07 5:15 PM 

REACHED 
0.0010 inch 

0.1000 inch 
0.9260 





2.33 

0.0004 


inch 

inch 


CASE 1 - 90/95 Xpod is NOW VALIDATED from Xpod to 
XL. An alternate 90/95 Xpod is available if Xpodpot or 
Optimum Xpoh (If listed) is also satisfied. 


Survey /Optimum Xpoh = 


0.000 inch [need Samples) 


NTlAC 90% POD - @ 

NTTAC 90/95 POD = @ 

False Call Rate = with UCL @ 95% = 

Largest Classlength , XL - 0.200 

Samples Needed g XL = 0 

Classlength Mid-point , Xm = 0.150 

Samples Needed @ Xm - 
Smallest Classlength, Xs = 

Samples Needed g Xs = 

New Smaller Classlength, Xss - 
BestLCL Classlength, Xlcl = 

Samples Needed eg Xlcl = 

POH Classlength, Xpoh = 

Samples Needed @ Xpoh - 
New Largest Classlength , 2XL = 

Xm is Near Verification Point = 

Opt. POD classlength, Xpodopt - 0.0950 

Samples Needed gXpodopt - 29 


inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 

inch 


O Probability of Hit (POH) fn Class Range ▲ Actual Lower Confidence Value (95%, F-distributi on) 


Esti mate of POH - Marquardt Gradi ent/Gri d X 


Hit/Miss 
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False Calls 


• False Calls are handled similarly except the upper confidence limit* P u 


_ , _ ,, n Number of False Calls (X) 

False Call Rate = - — - 

Number of False Call Opportunities (N) 


p (A- + l)F g (/„/ 2 ) 

" (N-X) + (X + l)F a (f l ,f 2 ) 


r/,=2(x+i) l 
I/2 = 2{N -X)\ 


is used 


• 95% Chance that the false call rate is less than or equal to the P u 
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False Calls 


• Test samples with no flaws present may be included in DOEPOD for determination of 
false call rate and the upper confidence value of the false call rate at 95% confidence. 
For test samples with no flaw present, enter flaw size of 0.00001” 

• False call rate may be explored and optimized by adjusting signal amplitude 
threshold. 

• Warning: May reach 90/95 X P0D at cost of increasing false call rate. Need to know 
what false call rate is acceptable. 

• False calls rate should not be accepted as is without first addressing the cause of the 
false call and identifying procedures to remove false calls. May need to modify or add 
inspection protocols. 


DOEPOD Data Entry 
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Hit / Miss Data 


ID Number 

CRACK SIZE (inches) 

DEPTH 

HIT/MlSS (0 or 100} 

Signal Amplitude 
Measured (Arbitrary 
Units} 

SIGNAL 

TREASHOLD 


0.342 


100 




0.251 


100 




0.242 


100 




0.213 


100 




0.203 


100 




0.199 


100 




0.133 


100 




0,134 


0 




0.169 


100 




0.166 


0 




Signal Amplitude Data 


ID Number CRACK SIZE (inches} 

DEPTH HIT/MISS (0 or 100} 

Signal Amplitude 
Measured (Arbitrary 
Units} 

SIGNAL 

TREASHOLD 

0.342 


0.5 

0.025 

0.251 


0.4 


0.242 


0.3 


0.213 


0.2 


0.203 


0.1 


0.199 


0.05 


0.136 


0.025 


0.134 


0.01 


0.169 


0.025 


0.166 


0.01 



DOEPOD REQUIREMENTS 
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• Inspection processes are to be under control and fixed. 

• Multiple inspection processes may be used on the same set of test samples with the constraint that 
Directed DOE POD is to be executed for each process separately. When multiple inspection processes 
or systems are used, the resulting directed sample requirements may be overlapping. In this situation, 
the user is to keep the non-overlapping directed sample requirements applied to the appropriate 
inspection process, while utilizing overlapping directed sample requirements for the multiple processes in 
order to minimize the number of generated test samples. 

• There are to be an equal number of unflawed samples during any test. 

• There are to be more than two (2) samples at different class lengths. 

• A moving class width that groups flaws of similar size is used to optimize the lower confidence value. 
This moving class width and the best lower confidence bound (value) optimization will be invoked if there 
are more than four (4) samples at different class lengths. 

• Flaw sizes must be greater than 0.00001” 

• Test samples with no flaws present may be included for determination of false call rate and the upper 
confidence value of the false call rate at 95% confidence. For test samples with no flaw present, enter 
flaw size of 0.00001” 

• The maximum number of test samples is 1 999. 

• Be prepared to generate, inspect, and evaluate test samples during the NDE technology capability 
determination. 

• Validated 90/95 X P0D is obtained when the user has reached and satisfied the sample requirements of 
Case 1 . That is, there is a 95% chance that the probability of detection of the system is greater 
than 90% for class lengths in the range 90/95 X P0D to XL. 
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SUMMARY 

• Concept for Binomialization of Test Data 

• Process for determining observed probability of hit (POH) and associated confidence limits 

• Utilization of moving class width to group flaws and for flaw class width optimization 

• Identification of POD CASES and directed actions needed to validate inspection systems. 

• False call rate and confidence 

• DOEPOD Data Entry 

• DOEPOD Beta (2,500 lines of code, PC and MAC, limited distribution) 

• Future work: DOEPOD upgrades 

> Interface with predicted POD MIL-HDBK-1823 

- companion tool 

> Address very limited data sets when 90/95 X P0D 
can never be reached, and communicating those risks. 
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• DOEPOD Probability of Hit estimating curve is for visualization only 
and not used in the DOEPOD analysis. 

• Probability of Hit estimating curve is not to be used for validation or for 
justification of validation. 

• The default function used in DOEPOD is: 

POH = Exp(a(1) + a(2) * Log(xl)) / (1 + Exp(a(1) + a(2) * Log(xl))) 

a(1) = a 

a(2) = /} 
xl = Flaw size 

• Other multi-parameter functions may be used 


